What Bayes Has to Say about the Evidence Procedure 3

The \evidence" procedure for setting hyperparameters is essentially the same as the techniques of ML-II and generalized maximum likelihood. Unlike those older techniques however, the evidence procedure has been justiied (and used) as an approximation to the hierarchical Bayesian calculation. We use several examples to explore the validity of this justiication. Then we derive upper and (often large) lower bounds on the diierence between the evidence procedure's answer and the hierarchical Bayesian answer, for many diierent quantities. We prescribe a simple, easy to compute, test that can check the validity of the approximation after the fact. We also touch on subjects like the close relationship between the evidence procedure and maximum likelihood, and the self-consistency of deriving priors by \\rst-principles" arguments that don't set the values of hyperparameters. \... any inference must be based on strict adherence to the laws of probability theory, because any deviation automatically leads to inconsistency."-S. Gull, in 5] \(Some have) estimated alpha from the data and then proceeded as if alpha is known. It is better to use the standard methods of Bayesian statistics and integrate out alpha."-B. D. Ripley, in 13]